[Data Validation] Add table configuration by aasthabharill · Pull Request #4188 · GoogleCloudPlatform/DataflowTemplates

aasthabharill · 2026-08-28T12:27:38Z

b/514940857

Added two new pipeline parameters:

tables: comma-separated list of source tables for small configurations
tableConfigurationFilePath: GCS File path containing list of source tables in a text file for large workloads

Design Decisions

Why two different template params?

tables will be extremely inconvenient to configure in case user needs to configure large number of tables

`tableConfigurationFilePath` is JSON file

Extensible for future additions
Example:

{
  "tableNames": ["table1", "table2"],
  "tableConfigurations": {
    "table2": { "columnsToValidate": [...] }
  }
}

Table names are to be specified as per their source names

More intuitive for users as they are more familiar with their source schemas. In case of Spanner table renames, the user specifies an overrides file/session file - which will be used to map their configured tables to their corresponding Spanner names

Fail fast if both template params are provided

There should be only one source of truth for the configured tables and since the purpose for both template params is the same, it doesn't make sense to pass both.

No mechanism to fail fast if configured tables do not exist in source

Since our source is the GCS Avro directory, if a source table exists but has zero rows, it will not have any entry in GCS.
Therefore, GCS records for a table not existing could mean either of two things:

The table doesn't exist on source and was misconfigured by the user
The table does exist on source but has 0 rows.
Since there's no way to differentiate between these two cases, we cannot introduce a fail-fast loop to identify a misconfigured table list.

Testing

Added comprehensive unit tests for all changes
Added an integration test for the scenario where tables mentions a subset of tables which were renamed in spanner (so tests table config in combination with overrides)

gemini-code-assist · 2026-09-02T11:07:47Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces table-level filtering for the GCS-to-Spanner Data Validation pipeline. By allowing users to specify a subset of tables via a comma-separated list or a GCS file path, the pipeline can now perform targeted validation, significantly improving efficiency when working with large schemas.

Highlights

Table Filtering Configuration: Added new template parameters 'tables' and 'tableListFilePath' to allow users to specify a subset of tables for validation.
Pipeline Integration: Implemented a new 'TableSelectionConfig' class and integrated it into the source and Spanner reader transforms to filter data processing.
Testing and Documentation: Updated documentation and added comprehensive unit and integration tests to ensure the new filtering logic functions correctly.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces table-based filtering to the GCS to Spanner Data Validation pipeline, allowing users to specify a subset of tables to validate via a comma-separated list or a GCS file path. Feedback on the changes focuses on improving robustness and code quality, including adding defensive null checks to prevent potential NullPointerExceptions in SourceReaderTransform, TableSelectionConfig, and CreateSpannerReadOpsFn. Additionally, it is recommended to use StandardCharsets.UTF_8 instead of string literals, and to remove a redundant Thread.sleep call in the integration tests as the Dataflow runner's startup time is already sufficient to satisfy the staleness window.

codecov · 2026-09-02T11:19:54Z

Codecov Report

❌ Patch coverage is 84.61538% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.69%. Comparing base (1576166) to head (c004f0f).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...e/cloud/teleport/v2/config/TableConfiguration.java	87.23%	0 Missing and 6 partials ⚠️
...oud/teleport/v2/config/TableConfigurationFile.java	16.66%	5 Missing ⚠️
...gle/cloud/teleport/v2/config/TableLevelConfig.java	0.00%	1 Missing ⚠️
...cloud/teleport/v2/dofn/CreateSpannerReadOpsFn.java	91.66%	0 Missing and 1 partial ⚠️
...ogle/cloud/teleport/v2/templates/GCSSpannerDV.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #4188       +/-   ##
=============================================
+ Coverage     35.85%   63.69%   +27.84%     
- Complexity      713     3011     +2298     
=============================================
  Files           250      576      +326     
  Lines         17151    33729    +16578     
  Branches       1755     3812     +2057     
=============================================
+ Hits           6149    21485    +15336     
- Misses        10487    11156      +669     
- Partials        515     1088      +573

Components	Coverage Δ
spanner-templates	`84.45% <84.61%> (∅)`
spanner-import-export	`∅ <ø> (∅)`
spanner-live-forward-migration	`88.88% <ø> (∅)`
spanner-live-reverse-replication	`80.80% <ø> (∅)`
spanner-bulk-migration	`88.95% <ø> (∅)`
gcs-spanner-dv	`87.99% <84.61%> (∅)`

Files with missing lines	Coverage Δ
.../teleport/v2/transforms/SourceReaderTransform.java	`100.00% <100.00%> (ø)`
...teleport/v2/transforms/SpannerReaderTransform.java	`100.00% <100.00%> (ø)`
...gle/cloud/teleport/v2/config/TableLevelConfig.java	`0.00% <0.00%> (ø)`
...cloud/teleport/v2/dofn/CreateSpannerReadOpsFn.java	`94.11% <91.66%> (ø)`
...ogle/cloud/teleport/v2/templates/GCSSpannerDV.java	`16.66% <0.00%> (ø)`
...oud/teleport/v2/config/TableConfigurationFile.java	`16.66% <16.66%> (ø)`
...e/cloud/teleport/v2/config/TableConfiguration.java	`87.23% <87.23%> (ø)`

... and 454 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

manitgupta · 2026-09-03T03:20:58Z

tableListFilePath is flat text file
Very easy to make, maintain and edit for customers as it's just a list of tables. JSON for this would be overkill and would require formatters for no real advantage.

I don't think it should be flat text file, I think it should be a JSON. Reason - extensibility.
A TableConfiguration can have more aspects about a table such as which columns to include/exclude etc. and keeping a JSON (backed by a POJO in the code) will keep this configuration flexible and extendible.

manitgupta

I haven't looked at the full PR yet, lets align on the existing comments and then will look at the rest

manitgupta · 2026-09-03T10:03:18Z

Can you put a sample tableConfiguration JSON in the description? I would like to have a look once

pull-request-size Bot added the size/L label Aug 28, 2026

aasthabharill added the addition New feature or request label Aug 28, 2026

pull-request-size Bot added size/XL size/XXL and removed size/L size/XL labels Sep 1, 2026

aasthabharill marked this pull request as ready for review September 2, 2026 11:04

aasthabharill requested a review from a team as a code owner September 2, 2026 11:04

aasthabharill requested review from jsuhani-2026 and shreyakhajanchi September 2, 2026 11:04

gemini-code-assist Bot reviewed Sep 2, 2026

View reviewed changes

aasthabharill requested a review from manitgupta September 2, 2026 11:42

manitgupta requested changes Sep 3, 2026

View reviewed changes

aasthabharill added 14 commits September 3, 2026 12:49

initial working changes

8993921

draft 2

fe4c7c4

testing

6ad5f6a

clean

f3d9f70

TableSelectionConfig

81b0ab4

final changes

95eb279

gemini-review

ce1ebfe

codecov

4df885a

it change

8cafb62

it change

5c07e13

it change

c323e79

json changes+rename

8f0b959

remove readme changes

6956647

terraform + rename

9323e2f

aasthabharill force-pushed the dv-table-config branch from 92afbd3 to 9323e2f Compare September 3, 2026 08:58

final touches

9892048

aasthabharill and others added 5 commits September 10, 2026 10:46

merge

b352d66

Merge branch 'main' into dv-table-config

a5a6f91

json changes + fix tests

f21411b

merge

b5d9828

Merge branch 'main' into dv-table-config

ccf36e8

manitgupta requested changes Sep 10, 2026

View reviewed changes

Comment thread ...-spanner-dv/src/main/java/com/google/cloud/teleport/v2/transforms/SourceReaderTransform.java Outdated

SourceReaderTransform change

c004f0f

manitgupta approved these changes Sep 11, 2026

View reviewed changes

aasthabharill merged commit d5f1c05 into main Sep 13, 2026
31 of 32 checks passed

aasthabharill deleted the dv-table-config branch September 13, 2026 09:53

Navigation Menu

Uh oh!

[Data Validation] Add table configuration - #4188

[Data Validation] Add table configuration#4188
aasthabharill merged 21 commits into
mainfrom
dv-table-config

aasthabharill commented Aug 28, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Sep 2, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 2, 2026 •

edited

Loading

Uh oh!

manitgupta commented Sep 3, 2026

Uh oh!

manitgupta left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

manitgupta commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aasthabharill commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Design Decisions

Why two different template params?

tableConfigurationFilePath is JSON file

Table names are to be specified as per their source names

Fail fast if both template params are provided

No mechanism to fail fast if configured tables do not exist in source

Testing

Uh oh!

gemini-code-assist Bot commented Sep 2, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

manitgupta commented Sep 3, 2026

Uh oh!

manitgupta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

manitgupta commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aasthabharill commented Aug 28, 2026 •

edited

Loading

`tableConfigurationFilePath` is JSON file

codecov Bot commented Sep 2, 2026 •

edited

Loading